-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Generalize handling of chunked array types #7019
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
for more information, see https://pre-commit.ci
…rray into cubed_integration
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚀
This reverts commit cdcb3fb.
for more information, see https://pre-commit.ci
…rray into cubed_integration
…warg to map_blocks
Co-authored-by: Illviljan <[email protected]>
for more information, see https://pre-commit.ci
…rray into cubed_integration
@Illviljan thanks for all your comments! Would you (or @keewis?) be willing to approve this PR now? I would really like to merge this so that I can release a version of xarray that I can use as a dependency for cubed-xarray. |
Thanks @TomNicholas Big change! |
Woooo thanks @dcherian ! |
👏 Congrats @TomNicholas on getting this in! Such an important contribution. 👏 |
Thanks for all your hard work on this @TomNicholas! |
Initial attempt to get cubed working within xarray, as an alternative to dask.
whats-new.rst
api.rst
I've added a
manager
kwarg to the.chunk
methods so you can doda.chunk(manager="cubed")
to convert to a chunkedcubed.CoreArray
, with the default still beingda.chunk(manager="dask")
. (I couldn't think of a better name than "manager", as "backend" and "executor" are already taken.)At the moment it should work except for an import error that I don't understand, see below.Fro cubed to work at all with this PR we would also need:
Array
vscubed.CoreArray
cubed-dev/cubed#123apply_gufunc
[WIP] apply_gufunc cubed-dev/cubed#119 - implemented in Addapply_gufunc
cubed-dev/cubed#149 🥳To-dos for me on this PR:
xarray.apply_ufunc
throughcubed.apply_gufunc
instead of dask'sapply_gufunc
when appropriate,from_array_kwargs
to opening functions, e.g.open_zarr
, andopen_dataset
,from_array_kwargs
to creation functions, such asfull_like
,store_kwargs
as a way to propagate cubed-specific kwargs when savingto_zarr
.To complete this project more generally we should also:
cubed.apply_gufunc
support multiple output arguments Support multiple outputs inapply_gufunc
cubed-dev/cubed#152cubed.unify_chunks
to matchdask.array.core.unify_chunks
xarray.map_blocks
to work on cubed arrays, ideally by first rewriting xarray's implementation ofmap_blocks
to usedask.array.map_blocks
cc @tomwhite